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1 Summary 



We consider the complexity of the LLL HNF algorithm ( |[HMM]| ). This al- 



gorithm takes as input an m by n matrix G of integers and produces as 
output a matrix b G GL^Z) so that A = bG is in Hermite normal form 
(upside down). The analysis is similar to that of an extended LLL algorithm 
as given in [ |vdK| . 

2 Sketch of the argument 

Let B be the maximum of the entries of QQ trans P° se . The main issue is 
whether we can estimate the entries of b in terms of B, m, n during the 
algorithm. The entries of A can then be estimated through A = bG. As they 
do not affect b, we may remove from G all columns that do not contribute 
a pivot to the Hermite normal form. Once that is done, A has as many 
columns as its rank, and at the end of the algorithm the product of its pivots 
is the covolume of the lattice spanned by its rows. This covolume can be 
estimated in terms of a minor of G, which by Hadamard is at most B rank ' 2 ''. 
This replaces the estimate di < B l of [ [LLL|| . As in [ |vdK]| we use the ordinary 



Euclidean inner product ( , ) for the rows of b, but also an inner product 
(v, w) = (Gv, Gw). (For the new G which has a rank equal to its number of 
columns.) The vectors v with (v,v) = are called isotropic. We let pr iso be 
the orthogonal projection according to ( , ) of M. m onto the isotropic subspace 
and put 

(v,w) mix = (pr iso v,pr iso w) + (v,w). 



1 



One estimates the ratio between (v, v) and (v, v) mix and one estimates 
(bi, 6j) m i X for any row bi of b. The required estimate of the entries of b follows 
from this, at least at the end of the algorithm. The algorithm computes a Her- 
mite normal form first for the top kmax rows of G, starting with kmax = 1, 
and increases kmax in steps of one. Each time just before one wants to in- 
crease kmax the analysis sketched above applies. Right after one wants to 
increase kmax we enter a stage which we will emulate with a procedure called 



trickledown, which we analyze as in | vdK |. It is followed by an ordinary 
LLL stage and then we get back to increasing kmax. Therefore no accidents 
happen. 



3 The analogy with an extended LLL algo- 
rithm 

Let ei,. . . ,e m be the standard basis of M. m . The Gram matrix gram = 
((ej, ej))™ =1 belongs to a positive semidefinite inner product ( , ) on M. m . 
Note that gram has integer entries. Let rank be the rank of G and assume 
that we have removed from G the columns that do not contribute a pivot. 
(In this paper a pivot is an entry of A that is the first nonzero entry in its row 
and also in its column.) Put isodim = m — rank and let b* denote the i-th 
Gram-Schmidt vector in the following sense. We have b* G (bi + Y^jLx 
and if 1 < j < i, j < isodim then (&*, bj) = 0, but if 1 < j < i, j > isodim 
then (b*,bj) = 0. With those notations the output satisfies: 

1. The first isodim rows bi of b are isotropic. 

2. With respect to ( , ) the first isodim rows of b form an LLL reduced 
basis of z^j=i 

3. The last rank rows of b form a basis of the lattice they span, and this 
lattice contains no nonzero isotropic vector. Furthermore, \(b*,bj)\ < 
(&*, bi) for isodim + l<i<j<m. 

4. For 1 < % < isodim, i < j < m we have \(b*,bj)\ < 1/2(6*, 6^). 
Remark 3.1 These properties are very much in the spirit of ||LLL|| and will 



thus allow us to estimate the (b i: bi) mix in a traditional manner. Of course the 
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output satisfies more properties, as actually bG has Hermite normal form, 
but we suppress that now. 



4 Stages 

Now that we have a way to look at the final result, let us discuss how we 
view things along the way. The moment that k wants to go beyond kmax 
is special. (As in |C[ we use kmax to denote the maximum value that k has 
attained.) At this moment one can estimate everything in the same manner 
as at the end. It is followed by a stage which we emulate by the procedure 
trickledown, and which ends when a new pivot or a new isotropic vector 
appears in the actual Havas, Majewski, Matthews LLL Hermite Normal Form 
algorithm. We then have an estimate of b as in | |vdK| |. After this one basically 



runs an ordinary LLL algorithm for the inner product ( , ) m ; x until k wants 
to go beyond kmax again. 

What makes it all rather technical is that ( , ) m j x always depends on which 
pivots and which independent isotropic vectors have been found. During 
trickledown one needs to take into account that one is dealing with an 
MLLL in the sense of [0, with the added complication that it is an extended 
MLLL algorithm in that one also requires the transformation matrix b. It is 
the latter which makes that one can not refer to H for the analysis. 



5 Running LLL 

In the definition of the inner product ( , ) we work with a G from which 
all columns have been removed where no pivot has been found yet. In the 
ordinary LLL stages we will have, 

• An integer matrix b of determinant one, 

• Integers k, kmax, 1 < k < kmax < m, 

• An integer isodim > 0, so that the first isodim rows bi of b span the 
isotropic subspace of Y^lj=i 

(Initialize with k = kmax = 1 and isodim = 0.) 
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Let pr iso be the orthogonal projection according to ( , ) of IR m onto 

E isodim 7™ j i , 

• =1 JROj and put 

(v,w) mix = (pr iso v, pr iso w) + («,«;). 
Let /ijj- be defined for i > j so that 

i-1 

& * = & i +^2^i,3 b j- 

The first standard assumption is then that, with respect to ( , ) m j x , the 
first k — 1 rows of b form an LLL reduced basis of X^=i except that one 
does not require 

+ ^ ji _ 1 6*_ 1 |^ ix > 3/416^1^ 

when % > isodim, and that the usual condition \fiij\ < 1/2 is weakened to 
l/^ijl < 1 f° r J > isodim. And the second standard assumption is that, as in 
JCJ, the first kmax rows of b form a basis of J^*™^ Ze^. 

We run the LLL algorithm with respect to ( , ) mix , except that one 
leaves out many swaps. (From now on we suppress mentioning the annoying 
weakening of the condition on the fiij.) Leaving out swaps is harmless, as the 



size estimates in ||LLL|| for the //y etcetera do not require that one executes 
a swap whenever such is recommended by the LLL test. Running LLL with 
respect to ( , ) m j x roughly amounts to running two LLL algorithms, one for 



and one for ( , ). That is why the pseudo-code in ]HMM] makes the 



distinction between coll = n + 1 and coll < n. One runs LLL until k tries to 
go to kmax + 1. If kmax = m we are through. If kmax < m then one should 
realize that because of the removal of columns from G the row Ckmax+iG will 
be dependent on the earlier ones. So we enter an extended MLLL, which we 
emulate with trickledown. 



6 Estimates 



We want to give estimates by changing [|vdKf| minimally. Thus let B > 2 so 



that the entries of gram are at most B. Our main result is 

Theorem 6.1 All through the algorithm all entries have bit length 
0(m log(mS)). 
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We do not care about the constants in this estimate. We leave to the reader 
the easy task of estimating the number of operations on the entries in the 
manner of [CLE]. One finds that 0((m + n) 4 log(mB)) such operations will 
do. 



6.2 Determinants 

Let gram mix be the Gram matrix ((e^, ej) m i x ) with respect to ei, . . . , tkmax- 
Its entries are at most B + 1. With Hadamard this gives 

\det(gram mix )\ < (V^(B + l)) m 

and the same estimate holds for its subdeterminants. We claim that the 
determinant of gram mix is an integer, so that we also get this upper bound 
for the entries of gram^ ix . To see the claim, consider as in |P|] the inner 
product ( , ) e given by (v,w) e = e(v,w) + (v,w). Its Gram matrix has a 
determinant which is a polynomial det e of e with integer coefficients. One 
may also compute det e with respect to a basis which is obtained from e^, 
■ ■ ■ > e-kmax through an orthogonal transformation matrix. By diagonalizing 
the Gram matrix of ( , ) we see that det(grom mix ) is the coefficient of e lsodim 
in det e . □ 

Lemma 6.3 For v G W 71 one has 

0,f)mix < m(B + l)(v,v) 
and for v E J^f^T one has 

(v,v) <m(VE(B + l)) m M) mix . 

Proof 

The supremum of { (v, v) mix \ (v,v) = 1 } is the largest eigenvalue of the 
gram matrix of ( , ) w j x with respect to ei, . . . , e m . The largest eigenvalue is 
no larger than the trace of this matrix. So it is at most m(B + 1). Similarly 
the largest eigenvalue of gram^ x it is at most m(y / m(i? + l)) m . □ 
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6.4 Vectors 

Now put 

i 

di SOi = l[(b*,b*) 
3=1 

for i < isodim and 

i 

+isodimi ^j+isodim) 

3=1 

for % < rank. As far as di is concerned we may compute modulo isotropic 
vectors, or also with ( , ) mix . Indeed 

(h* h* ) = (b* b* ) ■ 

\ j+isodimi j+isodim/ \ j+isodimi j+isodim /mm. 

for 1 < j < rank. Both disoi and dj are integers and they descend when 
applying LLL. (Throughout we assume familiarity with the arguments in 
JLTTJ.) In fact the (b* +isodim ,b* +isodim ) are themselves squares of integers. 
(Squares of the pivots of the moment.) 

One may also compute det (<?ram mix ) with the b* basis, as the transition 
matrix has determinant one. From that one sees that it is just disOi SO dimd ra nk- 
So we get disOi SO dim < (\frn(B + l)) m . In fact, for % < isodim one has the 
same estimate 

disoi < + l)) m 

because i was equal to isodim earlier in the algorithm and LLL only makes 
disoi go down. We have that 



rank 



d. t <B 

because this is so when a pivot is created and LLL only makes it descend. 
(Recall that the pivots are integers whose product is at most B rank ^ 2 , while 
dj is a product of some squared pivots (b* +isodim , b* +isodim ). The trickledown 
part also makes pivots descend.) 

Lemma 6.5 Let 1 < i < kmax. Then 

(vM5 + l))- m < (&*, 6*)mix < (V^(B + l)) m 

and if C > 1 is such that |^| 2 < C for 1 < j < i then 

(b u b t ) mix <mC(V^(B + l)) m 
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Proof 



Use the estimates of disoi, df. □ 

6.6 Preserved estimates 

Put C = (AmBf m . 

Lemma 6.7 The following estimates hold between applications of 
trickledown (each time k changes) 

1. disOi < (^Jm(B + l)) m for i < isodim, 

2. d{ < B rank for i < rank, 

3. (bi, bi) mhl < mC{^i{B + l)) m for i ^ k, 
I (6 fc , & fc )mix < m 2 9 m C{^i(B + l)) 3m ; 

5- < 1 for 1 < j < i < k, 

6. \{JLk,j\ < y n ~ k \fmC(^{B + l)) m for 1 < j < k, 

7. < V^n~C(V™( B + !)) m forl<j <i> k. 

Proof 



That these are preserved under LLL follows as in |[LLL|1 , so one has to check 
that they hold right after trickledown. Given our earlier estimates, this will 
be straightforward once we have shown that, at that moment, I 2 < C. 
Note that one could insert steps in the algorithm to reduce to the case C = 1 
instead of the outrageously pessimistic C = (4mB) 5m . □ 

7 Description of trickledown 

Before we can do estimates concerning trickledown we must describe it. 
One starts with having k = kmax + 1 < m. (So we look at the moment that 
kmax should be increased.) Consider the lattice generated by bi, . . . , bk ma x+i 
where b kmax+ i = e kmax+1 . As e kmax+1 G is dependent on the earlier rows of G 
now, this lattice contains a nonzero vector v with (f,f) m i x = 0. Modulo WLv 
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the vector b k is linearly dependent on the 6j with i < k. Changing the basis 
of 1*bk-i + we can achieve that modulo M.v the vector bk-i is linearly 
dependent on the 6, with i < k — 1. Then lower by one and repeat until 
k = isodim + 1, where isodim is the one from before the present trickledown. 
Or stop when the Havas, Majewski, Matthews LLL Hermite Normal Form 
algorithm produces a new pivot. If bk = b is0( a m +i is itself isotropic we increase 
isodim by one and pass to a new ( , ) mix . If a new pivot has been created we 
add back the relevant column to G and again pass to a new ( , ) m i x . This 
describes trickledown. 

One may worry about the fact that trickledown does not trace the 
Havas, Majewski, Matthews LLL Hermite Normal Form algorithm faithfully. 
We are close enough though. (And our replacement is has worse estimates 
than the original.) We are just leaving out some size reductions and we 
are taking together some swaps and reductions that make up the required 
change of basis of Z5fc_i + Z&&. The change of basis is the one coming from an 
extended euclidean algorithm. Thus we will further ignore that trickledown, 
which we took from |[vdK|| , does not quite trace this stage of the Havas, 
Majewski, Matthews algorithm. We simply blame their algorithm. 



8 Estimates during trickledown 

We look in more detail. Upon entering trickledown we freeze the old 
isodim, kmax and the b*, even though the bi will change. We also do 
not change ( , ) m i x . Let /i^o stand for (ek ma x+i,bi) and let /ijj stand for 
(b*, bi) m i x /(b*, 6*)mix if j > 0. Note that initially \fMjl < 1 for i < kmax, 
< j < kmax. We will estimate \fMjl as k descends. The key point is that 
we can also estimate //^o- This compensates for the fact that ( , ) m ; x is de- 
generate on Yl\=i X+l ^ e «- By combining //^o with ( , ) mix we will be able to 
estimate (bi,bi). It is to explain the estimate of /i^o that we prefer to work 
with trickledown. 

Say k > isodim + 1 and modulo Mv the vector is linearly dependent on 
the bi with i < k. Let us compute with b k , bk-i modulo V = Wv + Y^lZi 
We have bk = A*fe,fe— l^fc— i an d = b* k -i modulo V. With the extended 

euclidean algorithm of Q we find an integer matrix ( a ^ J of determinant 



7 S 

one so that ( a ^ ) f ' ) = ( 5*, | where r k is the index of Z in 
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Z + Zuz,i, i. More specifically, one has ( ^ ^ ) ( ^, ) — ( ' ) 
' \-7 a)\-l/r k ) \fi k , k -ij 

so (3 = r k and a = -r k jik,k-l- By we have | -^1 < \^k,k-iT k \ and \S\ < r k . 

(Actually this is wrong. Indeed only claims it when fi kyk -i is nonzero. 

We leave the modifications for the case /i/c.fc-i = as an exercise.) 

Now put Cfc_i = ab k _i + j3b k and c k = 7&fc_i + Sb k . The algorithm 

trickledown tells us to replace b k with c k and b k _i with c k ^\. We want 

to estimate the resulting new /i^-, which we call v^. For i different from k, 

k-1 nothing changes. Further \v k -i,j\ = \afi k -ij+Pfi k J < r k \fi k ^ 1 fi k ^ 1J \ + 

r k \fj, kjj \ and \v ktj \ = IjfMk-ij + ^kjl < nW^-iVk-ijl +r fc |^ fcj -|, which is the 

same bound. 

Lemma 8.1 v4s descends we have 
1- \fJ>ij\ < 1 f or k > i > j > 0, 

3- iMijI < 2 m ( v A B) ranfc+1 for k <i> j >0. 
Proof 

Initially we have fc = teas + 1 and |/ifc,j| 2 < -B- Now assume the estimates 
are true for the present k. We get < lMfc— i,i I + r fe|A i fcj| < 

2rfcmaxj which takes care of Now Ylk^isodim+2 r k is the ratio by 

which the covolume drops when adding e kmax+ iG to the lattice spanned by 
e x G, . . . , e kmax G. So it is at most (^fB) rank . Thus \v k ,j\ < 2 m (^/B) rank+1 . □ 

8.2 Bailing out 

When k has reached isodim + 1 or a new pivot has been created, it is time 
to forget the old ( , ) mix . But first use the estimates of the to estimate 

K Mmix < mA m B rank+1 (V^(B + l)) m 

and 
next 

(k - f i i , e hmax+li b t - ii lfi e kmax+l ) < m 2 (V^(B + i))2- 4 -+i jB ^+i 
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by means of Lemma 



, and finally 



(Mi) < (4m5) 4m 

say. 

Now update G, isodim, kmax, ( , ) mix . We have to compute the new /i^j. 
They can be estimated, as we have an estimate for (bj, bj) and for (b*, b*)^ ix . 
We get the estimate |/ij,i| 2 < (4m£>) 5m , which was needed in 
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